With the widespread use of web-based applications, and the Internet in general, concerns have been raised with the security of such servers and web applications operating on them in view of the array of bots and other web scraping potential malicious requests. Various security measures have been taken to combat these ever growing threats, including implementing web application firewalls (“WAFs”), such as the BIG-IP Application Security Manager™ (“ASM™”) product developed by F5 Networks, Inc., of Seattle, Wash., which may be used to analyze network traffic to Web application servers for identifying and filtering out malicious packets or to otherwise thwart malicious attacks.
Web scraping is a computer software technique of extracting information from web pages in which web scrapers, search engines or bots index web content and transforms unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Although safeguards against web scraping exist, such as CAPTCHA, bots and other web scrapers have become more complex and able to bypass these safeguards to take content off of web pages.